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Abstract 


Research on individual differences indicates that 
students vary in how they interact with and perform 
while using intelligent tutoring systems (ITSs). 
However, less research has investigated how individual 
differences affect students’ interactions with game-based 
features. This study examines how learning outcomes 
and interactions with specific game-based features (off- 
task personalization vs. on-task mini games) within a 
game-based ITS, iSTART-ME, vary as a function of 
students’ performance orientation. The current study 
(n=40) is part of a larger study (n=126) conducted with 
high school students. The analyses in this study focus on 
those students assigned to iSTART-ME. Results indicate 
that students with higher levels of performance 
orientation perform better during training, progress 
further within the system, and interact less frequently 
with off-task game-based features. These results provide 
further evidence that individual differences play an 
important role in influencing students’ interactions and 
achievement within learning environments. 


Introduction 


One focus of educational research is on student individual 
differences and how those differences impact the effects of 
certain pedagogical approaches. Research has shown that 
differences in students’ ability levels, attitudes, and 
motivation critically impact their interactions and 
performance within the classroom (Ames & Archer, 1988). 
Likewise, as educators increasingly integrate new 
technologies within the classroom, questions emerge 
concerning the role of individual differences in moderating 
the effects of educational technologies. For example, 
several researchers have focused on students’ engagement 
and style of interactions with intelligent tutoring systems 
(ITSs), and how those differences among students affect 
achievement and learning outcomes (e.g., D’Mello, Picard, 
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& Graesser, 2007; Baker et al., 2008; Jackson, Graesser, & 
McNamara, 2009). 


This study focuses on the effects of individuals’ 
performance orientation on interactions and learning 
within a game-based ITS. Performance orientation (also 
known as ego orientation or ego-involvement) is an 
attitude that emphasizes the importance of superior 
individual achievement compared to others on designated 
tasks (Ames, 1992). According to this perspective, students 
perceive learning as secondary to achievement and the 
recognition they receive from succeeding in the task 
(Ames, 1992). High levels of performance orientation in 
the classroom are negatively related to learning outcomes 
(see Pintrich, 2000, for a review). So, those students who 
are focused on achievement relative to others rather than 
the intrinsic benefits of learning do more poorly on 
learning tasks. 

One explanation for these negative learning outcomes is 
that students who exhibit high levels of performance 
orientation are more likely to divert attention and effort 
away from the learning task (Fisher & Ford, 1998). These 
students exhibit more concern about how _ their 
performance will measure up to their peers and seem to be 
less concerned with the learning goal. Therefore, they are 
more likely to seek out shortcuts or loopholes, indirectly 
negatively impacting learning (Fisher & Ford, 1998). 

Taking shortcuts in an ITS environment has been 
referred to as gaming the system. Gaming is defined as 
getting through the system as successfully and as fast as 
possible (Baker et al., 2004), usually by finding loopholes 
and shortcuts. Baker and colleagues expected performance 
orientation to be positively related to the prevalence of 
gaming behaviors in an ITS. This hypothesis followed 
from the assumption that those who game a system are 
analogous to students in a classroom who show more 
concern with acknowledgment and less concern with 
learning. While this reasoning appears quite sound, alas, no 
significant relation has been found between gaming 
behaviors and performance orientation (Baker et al., 2008). 


Hence, the hypothesis that performance orientation is 
related to gaming the system has not held up. 

In the current study, we explore an alternative 
hypothesis. We propose that performance orientation is 
related to achievement and interactions in educational 
technologies, but not in the way expected by Baker and 
colleagues (2004, 2008). While students with high levels of 
performance orientation may not game the system per se, 
they may still interact with it differently than those with 
lower levels of performance orientation. For instance, if 
high performance oriented students are focused on 
outshining others, they may also focus their efforts on 
system features that promote incentives or rewards. This 
expectation is particularly relevant to game-based ITSs. 

Game-based features have been shown to increase 
student engagement and enjoyment within learning 
environments (Rai & Beck, 2012; Rai, Beck, & Heffernan, 
2010; Jackson & McNamara, 2011, 2012). Hence, there is 
an increased interest in incorporating game-based features 
into ITSs to enhance student engagement, particularly 
during extended training sessions (Cordova & Lepper, 
1996; McNamara, Jackson, & Graesser, 2009). However, 
while game-based features may enhance engagement, they 
may also act as seductive distracters (Rai, Beck, & 
Heffernan, 2010). Seductive distracters have the 
unfortunate consequence of pulling students’ attention 
away from the learning tasks and consequently decreasing 
performance (Harp & Mayer, 1998). 

Whether game-based features increase engagement or 
distract from learning may depend on the type of feature. 
Game-based features can be tangential to learning (off- 
task; e.g., avatar personalization or navigational choices) or 
an extension of the learning task (on-task; e.g., extended 
practice mini-games or practice quests). These features can 
also vary in the amount of incentives they provide based on 
student achievement within the system. Although research 
has investigated the impact of game-based features on 
enjoyment and learning, very little work has investigated 
why students choose to interact with specific types of 
features versus others. Understanding the influence of 
individual differences on system interactions is crucial for 
building adaptive systems that promote learning. 

In the current study, we investigate the relations between 
individual differences in performance orientation and 
system interactions, system achievements, and learning 
gains. Using the iSTART-ME system, we examine how 
students’ interactions with personalizable features (i.e., 
editable avatars, changeable background themes, and 
changeable pedagogical agents) and mini-games vary as a 
function of self-reported levels of performance orientation. 
In the context of iSTART-ME, we expected that high 
performance oriented students would be more likely to 
interact with features that provide incentives (e.g., trophies, 
currency, badges). In turn, by doing so, they would also be 
more likely to veer away from game-based features that do 
not overtly promote a measure of success. 
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iSTART 


Interactive Strategy Training for Active Reading and 
Thinking GGSTART) is an ITS that was designed to 
improve students’ comprehension of science texts through 
the instruction of reading strategies (McNamara, 
Levenstein, & Boonthum, 2004). This system is divided 
into three modules: introduction, demonstration, and 
practice. During the introduction module, students interact 
with three pedagogical agents who discuss definitions and 
show students examples of reading comprehension 
strategies. After the introduction module is completed, 
students are transitioned into the demonstration module. In 
this model students watch as two pedagogical agents apply 
the previously introduced reading strategies to texts. 
Students are then asked to identify which strategy is being 
used by the agents in each text. Finally, the practice 
module of iSTART provides students with an opportunity 
to apply the strategies they have seen to science texts. In 
this module, students are shown two example texts and are 
asked to generate self-explanations for target sentences. A 
pedagogical agent provides formative feedback on 
students’ use of the comprehension strategies. An extended 
practice module provides the means for additional practice 
over weeks or months. 

A computational algorithm within iSTART assesses the 
quality of students’ self-explanations and guides the 
feedback that they receive from the system. Self- 
explanations are assessed on a scale of 0 to 3. A score of 
“0” is provided when students’ self-explanations are 
composed of irrelevant information or are too short. A 
score of “1” is given when self-explanations do not 
elaborate upon the provided information in the text. A 
score of “2” indicates that students’ self-explanations 
incorporate inferences from the text, and a“3” indicates 
that the self-explanation incorporates information about the 
text at a global level. 


iSTART-ME 


The practice module and extended practice module within 
iSTART enhance students’ ability to self-explain and 
comprehend challenging text; however, the repetitive 
nature of these modules can cause some students to 
disengage (Bell & McNamara, 2007; Jackson & 
McNamara, 2012). iSTART-ME (Motivationally 
Enhanced) expands upon the original isSTART system by 
incorporating educational games and adding game-based 
features into the interface of the original system (Jackson, 
Boonthum, & McNamara, 2009). This system incorporates 
the three modules from the original iSTART system: 
introduction, demonstration, and practice. However, an 
added module, called extended practice, was added to 
increase students’ engagement and enjoyment. In this 
module students interact with a game-based interface that 
was designed to reinforce the comprehension strategies. 
The iSTART-ME interface is controlled through a 
selection menu where students can choose to play games, 


read new texts, and interact with game-based features (see 
Figure 1 for menu screenshot). When students choose to 
interact with texts and play games, they earn points within 
in the system called iBucks. These iBucks serve as in- 
system currency and can be used to unlock the game-based 
features on the interface. 


iBucks (iB) 
3897 


My Character iSTART Level 
15 -- Ultimate Genius 


ses 


006Test060606 


You have 30 assigned text(s) 
Next text is "Energy Transformers" 


You must complete all assignments before spending iBucks 


Cost 
300 iff 
300 if 
300 if 
300 if 
300 i 
300 ifs 


Chonge background color 
‘Swap out the tutor 

SE Lifeline 
Balloon Bust 
Dungeon Escape 


Edit my character 
(hit, hair, headgear, glasses) 


Figure 1. Screenshot of iSTART-ME Menu 


Students can earn iBucks through three different 
methods of generative practice: Coached Practice, 
Showdown, and Map Conquest. Coached Practice is the 
form of generative practice from the original iSTART 
system. Showdown and Map Conquest are both game- 
based generative practice modules that use the same 
assessment algorithm as Coached Practice. All three of 
these game-based practice modules require students to 
generate self-explanations and practice applying the 
strategies (just as they do within Coached Practice). 

As students accumulate more iBucks, they advance to 
higher levels, where new features are unlocked. Levels 
range from 0 (starting level for all students) to 25 (the 
maximum level that students can achieve). Each level 
requires more iBucks than the previous in order to advance 
to a new level. This ensures that students must exert more 
effort as they progress through higher levels within the 
system. Students can use their earned iBucks to interact 
with different game-based features within the ITS 
interface. There are two primary types of game-based 
features that students can use points to interact with: mini- 
games and personalizable features. 

Mini-games were designed to add game-based practice 
for strategies that the student had previously seen within 
the iSTART-ME system. For instance, in Balloon Bust 
students are presented with a text and then provided a 
sample self-explanation. They must decide what strategy 
was used to generate each sample self-explanation by 
clicking on a balloon that represents each strategy (see 
Figure 2). Within mini-games, students can earn points and 
advance to higher iSTART levels based on_ their 
performance. Each mini-game play costs students 300 of 
their earned iBucks. 

Personalizable features were designed to engage 
students’ interest by allowing them to personalize and 
control their environment as a reward for performing well 
within the game-based learning tasks. Personalizable 
features include editable avatars, customizable background 
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themes, and interchangeable pedagogical agents. For 
instance, students can use their earned iBucks to change 
their avatar’s hair colors or clothing (see top left of Figure 
1). Each personalization action requires students to spend 
300 of the iBucks that they have earned. These features 
were designed as off-task incentives and do not contribute 
to their earned points or iSTART levels within the system. 


€ Click this button to pause 


Target 
Sentence 5 
Self - | The 
Explanation 


Figure 2. Screenshot of Balloon Bust 


Methods 


Participants in this study included 40 high school students 
from a mid-south urban environment. The sample of 
students included in the current work is a subset of 125 
students who originally participated as part of a larger 
study that compared three conditions: iSTART-ME, 
iSTART-original, and no-tutoring control. Our study 
focuses solely on the students who were assigned to the 
iSTART-ME condition. These students had access to the 
full system interface where the game-based features 
(personalizable features and mini-games) were available. 

All 40 students in the iSTART-ME condition completed 
an 11-session experiment that consisted of a pretest, 8 
training sessions, a posttest, and a delayed retention test. 
During the first session, participants completed a pretest 
including a self-report survey that assessed individual 
differences in affect, motivation, and attitudes toward 
technology and games. During the 8 training sessions, the 
iSTART-ME participants took part in the extended practice 
portion of the experiment. Each of these sessions lasted a 
minimum of | hour. At the beginning of each session, 
students completed a short questionnaire assessing their 
current mood and perception of the system. After this 
questionnaire, students were transitioned to the iSTART- 
ME interface, where they were free to interact with the 
system as they chose for an hour. During session 10, 
students completed a posttest, which included the same 
measures administered at pretest. Finally, approximately 1 
week after the posttest, each student returned to complete a 
retention test, which contained measures similar to the 
pretest and posttest (i.e., self-explanation and attitudinal 
measures). 

Students’ performance orientation rating was measured 
at pretest through a self-report survey in which students 


were asked to rate themselves and their attitudes toward 
educational technology. To assess students’ performance 
orientation, each student indicated the relative importance 
of the following statement, "It is important for me to do 
better than others within the system." This rating was on a 
scale from | (it is not important to me) to 6 (it is very 
important to me). This was the only measure of 
performance orientation in the pretest survey; therefore, the 
reliability of the measure could not be calculated. 
Students’ reading comprehension ability was assessed 
using the Gates-MacGinitie Reading Test (MacGinitie & 
MacGinitie, 1989). Self-explanations were scored using the 
automated iSTART assessment algorithm (McNamara et 
al., 2007). 


Results 


The current study investigated how _ performance 
orientation related to differences in student behaviors and 
achievements within iSTART-ME, a game-based ITS. 
Using the frequency of each student’s interaction with on- 
task and off-task features, we examined how individual 
differences influenced interactions within the system. In 
the current study, off-task behaviors included editing an 
avatar, customizing the background theme, or changing a 
pedagogical agent, whereas on-task behaviors included 
choosing to play extended practice mini-games. 
Additionally, we measured each _ student’s daily 
achievements through iSTART levels, points earned, and 
the quality of daily self-explanation scores within the 
system. Daily self-explanations were generated through 
students’ interactions with generative practice games and 
generative mini-games. 

In order to assess students’ system interactions, we 
computed the tendency of each student to spend their 
earned iBucks on game-based features by calculating the 
proportion of earned iBucks that each student spent on 
game-based features (total iBucks spent / total iBucks 
earned). This provides each student’s tendency to spend 
iBucks within iSTART-ME, which we refer to as 
spendency. We also calculated the proportion of iBucks 
that each student spent on personalizable features (iBucks 
spent on personalization / total iBucks spent) and the 
proportion of iBucks that students spent on mini-games 
(iBucks spent on mini-games / total iBucks spent). This 
provided the proportion of iBucks spent on both 
personalizable features and on mini-games. 

Pearson correlations were calculated between students’ 
levels of performance orientation, self-explanation quality, 
prior motivation to learn, study expectations, and system 
interactions. Correlations in this table were grouped by 
conceptual relevance. As shown in Table 1, seven variables 
were significantly correlated with performance orientation. 
We discuss those correlations in the order they appear in 
Table 1. 

Results indicated that there was no relation between total 
points earned and performance orientation. However, 
students who reported a higher level of performance 
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orientation tended to spend a lower proportion of their 
earned iBucks (i.e., they displayed lower spendency). 
Moreover, those with higher performance orientation who 
did spend iBucks, were less likely to spend iBucks on 
personalizable features (off-task, non-incentive earning 
features). Hence, students with high performance 
orientation levels were not earning more or less iBucks in 
the system; they were simply spending a lower proportion 
of their earned iBucks. Additionally, when these students 
did spend system currency they chose to interact less 
frequently with off-task features. 

The correlational results also revealed a significant 
positive relation between students’ performance orientation 
levels and their expected enjoyment, prior motivation to 
learn, and confidence that they could learn from the 
system. Overall, high performance orientation students 
were more likely to show positive perceptions toward the 
system. 

Finally, we also see that performance orientation is 
significantly related to iSTART levels and daily self- 
explanation quality. More specifically, students who 
reported higher levels of performance orientation tended to 
reach higher levels within the system and received higher 
scores on their self-explanations during daily training. 
However, performance orientation was not related to self- 
explanation scores at either posttest or retention test. 


Table 1 
Correlations between performance orientation and system 
interactions, perceptions, and achievements 


Dependent Measure Performance Orientation 


_TotaliPointseamed ae 
_Spendency ON. 
_Prop spenton mini-games 2D 2 es 
_Prop spent on personalization == 367% 
_Expected enjoyment G08 Fiettetetccs 
_Motivationtoleam O86 ee caee esas 
_ConfidentI canleam 809 ** 
STARTIevel 317* 
_DailySEscores BO oes 
_Posttest SE scores ee 062, 
Retention SE scores 302 


*p < 05; ** p< .01 


The current study investigated how performance 
orientation impacted students’ outcomes and interactions 
within the system. Results revealed that there was a 
significant relation between system interactions, system 


achievement, daily  self-explanation scores, system 
perceptions, and students’ levels of performance 
orientation. 

We were also interested in gaining a deeper 


understanding of the impact that performance orientation 
has on the system’s learning objectives (i.e., to improve the 
quality of self-explanations). Specifically we investigated 
the impact of students’ orientation on daily  self- 
explanation quality above and beyond students’ initial 
abilities and motivation at pretest. A regression analysis 


was conducted to examine how students’ pretest self- 
explanation scores, prior motivation, and self-reported 
performance orientation influenced daily self-explanation 
training scores. The three independent variables were 
entered as predictors of daily self-explanation scores in a 
forced-entry linear regression. The overall model for this 
regression was significant, F(1,36)= 4.617 p <_ .05, 
R*=.355. The first block of the regression (i.e., Model 1 in 
Table 2) shows that pretest self-explanation quality 
significantly predicts students’ ability to  self-explain 
during training; this is to be expected. Model 2 shows that 
students’ prior level of motivation to learn does not 
account for a significant amount of variance in daily self- 
explanation quality. Within Model 3, both pretest self- 
explanation scores (¢ = 3.73 p <.001) and performance 
orientation (¢t = 2.15, p <.05) were significant predictors of 
daily self-explanation scores. Pretest motivation was not a 
significant predictor (p>.05). Hence, performance 
orientation was a significant predictor of daily self- 
explanation quality above and beyond initial  self- 
explanation ability and prior motivation (with a unique R* 
of .083). 


Table 2 
Linear regression analyses predicting daily self-explanation (SE) 
score 


Variable B SE B AR? 

Model 1 263 ** 
Pre-Test SE 414 112 513** 

Model 2 .009 
Pre-Test SE 397 116 .492** 
Prior Motivation 048 071 .096 

Model 3 .083* 
Pre-Test SE 414 111 513** 
Prior Motivation -.024 .076 -.048 
Performance 122 057 320 * 
Orientation 

*p < .05; ** p< .01 

Conclusions 


iSTART-ME was designed as a game-based ITS that 
maintains student engagement over an extended time 
period while students practice applying self-explanation 
and comprehension strategies. Although game-based 
features have been shown to increase student engagement 
(Jackson & McNamara, 2011), little work has been 
conducted to investigate how individual differences may 
impact the frequency of student interactions with these 
features. Our study investigated this issue by examining 
how individual differences in performance orientation 
impacted the types of interactions that students exhibited 
within the iSTART-ME interface. We also examined how 
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this orientation affected daily system achievements 
throughout the eight training sessions. 

Results from the current study indicated that students’ 
self-reported performance orientation impacted their 
interactions with distinct features. Specifically, as ratings 
of performance orientation increased, students had a lower 
tendency to spend their earned iBucks. However, when 
they did spend iBucks, they were less likely to spend them 
on personalizable features. One reason for this behavior 
could be that these features provided the students with no 
additional earned incentives (i.e., points or iBucks). 
Performance oriented individuals’ focus on their own 
achievements compared to others, and these personalizable 
features do not indicate to students how well they are doing 
or give them any additional recognition. 

Students with higher levels of performance orientation 
were also more likely to reach higher iSTART levels 
within the system. iSTART levels are based on points 
earned within the system and are an outward measure of 
achievement that is displayed on the interface and could be 
used by students to gauge their individual performance in 
the system. The results from the current study suggest that 
individual differences in performance orientation are 
significantly related to students’ selection and success for 
tasks that outwardly reflect abilities within the system. 

In the current study the only measure of learning that 
was impacted by performance orientation was students’ 
average daily self-explanation score. As students’ ratings 
of performance orientation increased, so did their daily 
self-explanation score. Daily self-explanation scores come 
from students’ interactions with generative games, where 
performance is also rewarded with iSTART points. These 
results indicate that high levels of performance orientation 
led to stronger training self-explanation scores. However, 
despite these training effects, there was no relationship 
between students’ levels of performance orientation and 
their self-explanation scores at posttest or retention test. 
Consistent with prior research, these results may stem from 
the fact that high performance oriented individuals are 
more concerned with achievement (e.g., iSTART levels, 
points and trophies) than deep learning and retention. 

The current study demonstrates the importance of 
individual differences and their impact on ITS interactions 
and system achievements. In our study, performance 
orientation explained variance in students’ system 
interactions and daily system achievements. The level of 
students’ orientation also accounted for the quality of daily 
self-explanations over and above pretest self-explanation 
ability and prior motivation. 

Future investigations of individual differences may give 
us a better understanding of why some features are 
preferred by some students, but ignored by others. Future 
work should also be directed at investigating and 
identifying how individual differences may impact overall 
learning. Understanding how these differences impact the 
way in which students approach and interact with specific 
aspects of systems will allow us to continue to build 


interactive learning systems that adapt to students’ 
strengths and preferences. 
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